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ABSTRACT 

The New York State Education Department is currently 
developing an innovative and comprehensive system for teacher 
certification assessment. One component is the Assessment of Teaching 
Skills-Performance (ATS-P) , a video-based performance assessment for 
which teachers would prepare and submit a videotape of their 
instruction for evaluation by permanently certified New York State 
teachers. The development of the ATS-P is traced, and plans for its 
future application are discussed. An early feasibility study, 
conducted by the project developer, National Evaluation Systems, 
Inc., examined the basic technical issues by videotaping several 
public school teachers at work. This study indicated that basic video 
equipment used without special training could make tapes with a high 
level of quality. In a Phase 2 study, the ability of teachers to 
produce videotapes was ascertained. A third component of the 
feasibility study examined the ability of teachers to score 
videotaped performance consistently. The results of all three studies 
indicate the feasibility of using videotape recordings in teacher 
evaluation. Some preliminary policies for administration of the ATS-P 
have been developed. Although many revelopment activities remain, the 
ATS-P is on its way to providing a reliable assessment of teacher 
performance. (Contains 2 references.) (SLD) 



* Reproductions supplied by EDRS are the best that can be made * 

from the original document. * 
*Vc*****yc*yc*7vyc*yr*****yry:yfyryryr>vyr*^^ 



VIDEO-BASED TEACHER PERFORMANCE ASSESSMENT: 



INNOVATIONS IN NEW YORK STATE TEACHER CERTIHCATION TESTING 



Ottice of Educational Research and irr.provement 



U.S. DEPARTMENT OF EDUCATION 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



iVThis docgment has been reDroduced as 
'eceived from the person o» organization 
or:Qinating .t 



EDUCATIONAL RESOURCES INFORMATION 



CENTER (ERIC) 



C Minor Changes have been made to improve 

reproduction Quality 



• P0>ntS Ot vie'pV or opinions Stated «n this docu 

ment do nol necessarijy represent official 

OERI DOSitiOn or pOliCy 



John R. Silvestro 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) " 



National Evaluation Systems, Inc* 

Gerald L. Freeborne 
Edith Hunsberger 
Sanford E« Lake 
Charles C. Mackey, Jr. 
New York State Education Department 



Paper presented at the 1993 annual meeting of the 
American Educational Research Association 



Copyright® 1993 by National Evaluation Systems, Inc. (NES*). Ail rights reserved. 



"NES®/ "National Evaluation Systems, Inc.~, and die ''NES*'' logo are trademarks of 
National Evaluation Systems, Inc. 



AERATlAN.VVRO/032«t3 



Video-Based Teacher Assessment: 
Innovations in New York State Teacher CertiHcation Testing 



Introduction 

The New York State Education Department is current-y developing an innovative and 
comprehensive system for teacher certification assessment. Among the purposes of the New 
York State Teacher Certification Examinations is the desire to develop an integrated system of 
meaningful and authentic assessment for teaching certification candidates (The Commissioner's 
Task Force on the Teaching Profession, 1988; University of the State of New York, 
Regulations of the Commissioner of Education). Candidates for a provisional] New York State 
teaching certificate will be required to pass two assessments. One is the Liberal Arts and 
Sciences Test (LAST), which requires candidates to demonstrate conceptual and analytical 
skills, critical thinking and communication skills, and multicultural awareness. The other is 
the Assessment of Teaching Skills-Written (ATS-W), which is designed to determine if 
candidates possess the knowledge and skills of professional practice to teach satisfactorily in 
New York State public schools. Candidates for a permanent New York State teaching 
certificate will be required to pass two additional assessments. One is the Content Specialty 
Test (CST), an assessment of the content knowledge important to the job of a New York State 
public school teacher. The other is the Assessment of Teaching Skills-Performance (ATS-P), a 
video-based performance assessment for which teachers would prepare and submit a videotape 
of their instruction for evaluation by permanently-certified New York State teachers. This 
paper will explore the development of the ATS-P to date and current plans for the future 
development and implementation of this assessment. 



Initial Research and Development Activities 

The New York State Education Department is working with National Evaluation Systems, 
Inc., of Amherst, Massachusetts to develop the ATS-P. The work to date has been designed to 
be exploratory in nature, so that many alternative directions for the video-based assessment 
are being evaluated before final directions for the assessment are determined. The valuable 
input of many teachers, teacher educators, bias reviewers, and professional teacher 
associations has been sought and obtained in developmental efforts to date. 

An extensive video feasibility study was conducted as part of an early effort to examine the 
technical and logistical issues associated with teachers producing videotapes of their classroom 
instruction. This study had several key phases. In Phase 1, an effort was made to examine the 
basic technical issues involved in preparing videotapes of teadiing in actual elementary and 
secondary public school classrooms. Several public school teacher volunteers, who taught a 
variety of subjects at a number of grade levels, were videotaped for anywhere from 30 minutes 
to two hours. Different types of video recording cameras were used and different camera 
positions (e.g., front, side, and rear of classroom) were experimented with. The results of 
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this Phase 1 study indicated that by using basic school video equipment and without specialized 
training in video recording techniques, tapes with a high level of video quality could be 
produced. 

The success of the Phase 1 study led to a Phase 2 study. The goal of the Phase 2 study was to 
determine if public school teachers, with a written procedures manual, could set up video 
recording equipment and produce either by themselves or with the assistance of other school 
personnel, a videotape of approximately 30 minutes of their classroom instruction. A second 
goal of the Phase 2 study was to find out if teachers could produce videotapes that included 
both whole-group and non-whole-group (e.g., cooperative learning, group work with 
manipulatives, etc.) instructional segments. A third goal of the study was to examine the effea 
of video camera position (front of classroom versus rear of classroom) on overall impressions 
of teacher performance. Public school teachers, including elementary and secondary teachers 
from urban, suburban, and rural settings, participated in the study. 

The results of the Phase 2 study provided evidence that public school teachers could produce 
high quality videotapes of their instruction both by themselves and with the assistance of other 
school personnel. Some teachers set up their camera lens in a fixed position and taught within 
the camera angle of view, thus producing a self-recording. Other teachers had a colleague or 
student teacher operate the camera and had them pan the camera to follow the movement of 
the teacher and the students throughout the classroom (camera-operator recording). Teacher 
videotapes included both whole-group and non-whole^oup instruction. While in general the 
technical quality, in terms of videotape scorability, of the whole-group recordings was better 
than that of the non-whole-group recordings, it appears possible to draw conclusions about 
teachers' skills in interacting with students and managing classroom activities from recordings 
of both whole-group and non-whole-group mstruction. In terms of the use of a front-of-c!ass 
camera placement versus a rear-class camera placement, there were advantages to each. 
However, it does not appear that the position of the camera affects overall judgment*^ of teacher 
performance. 

The third component of the video feasibility study involved a preliminary effort to determine if 
a sample of the Phase 2 videotapes could be scored consistently by a group of experienced 
public school teachers. Toward this end, a preliminary scoring scale was developed based 
upon die following five ATS-P objeaives. These objectives had been previously validated by 
several hundred certified and practicing New York State public school educators and teacher 
educators by means of a job analysis survey. 

L Understand principles and procedures for organizing and implementing lessons and use 
this knowledge to help learners construct meaning and achieve intended outcomes. 

2. Understand multiple approaches to inssruaion and use this knowledge to facilitate learning 
in various situations. 

3. Understand how motivational principles and praaices can be used to promote studem 
achievement and active engagement in learning. 
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4. Understand how to use a variety of communication modes to promote student learning and 
to foster a climate of trust and support in the classroom. 

5. Understand how to structure and manage a classroom to create a climate that fosters a 
safe and produaive learning environment. 



The preliminary scoring scale used in the scoring tryout was a 3-point scale that was part of a 
focuspd. holistic scoring system. Scorers were asked to provide a single integrated judgment 
about the overall effectiveness of the videotaped teachers in light of each teacher's performance 
on the five validated test objectives using the following scale: 

3 Excellent teaching perfonnance 

2 Satisfactory teaching performance 

1 Inadequate teaching performance 



Participants in the scoring process were provided with approximately five hours of training in 
using the preliminary scoring scale, including opportunities to view and discuss their scores in 
a group setting. They were also provided with an opportunity to independently view and score 
additional videotapes. The results of the scoring tryout indicated that there was a high degree 
of consistency in the scores assigned by multiple scorers to the same videotapes of teacher 
instruction. Using the preliminary 3-point scale, there was near perfect a^eement among the 
scorers in their ratings of videotaped teacher performance. 



Videotape Program Summarizing Status of Video-Based Performance Assessment 

A videotape presentation has been prepared through the joint efforts of the New York State 
Education Department and National Evaluation Systems. This program was produced to 
provide New York State public school educators, teacher educators, test development 
committees and other important constituent groups with an overview of what a videotape 
submission by an ATS-P candidate would look like, how a group of experienced public school 
teachers could be trained to score the videotape submission, and discussions by the scorers on 
the degree of consistency of the scores that diey assign to a videotape using the preluninary 
focused holistic scoring system. This videotape is an important adjunct to this paper and will 
be shown as part of the presentation of diis paper at the American Educational Research 
Association annual meeting in April, 1993. 
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Preliminary Policies for the Administration of the ATS-P 

The New York State Education Department and National Evaluation Systems have had some 
initial discussions on the policies regarding the administration of the ATS-P. These 
preliminary policies will be reviewed, and revised as needed, based on the input of New York 
State public school teachers, teacher educators, and a panel of New York State educators who 
have experience with and are sensitive to, issues of potential bias. Current thinking about the 
administration of the ATS-P has led to the following general policy recommendations: 

• Candidate videotape submissions should not exceed 30 minutes in length, including a 
minimum of 10 minutes of whole-group instruction and 10 minutes of non-whole-group 
instruction. 

• Candidates will be free to make as many practice videotapes as they desire before selecting 
the videotape for their official ATS-P submission. 

• Candidate videotape submissions will be accompanied by a Context of Classroom 
Instruction Form, on which they will provide their instructional objectives, descriptions of 
previous instruction, and some background information on their students. This form may 
be reviewed by scorers prior to viewing each videotape submission. 

• Candidates will be encouraged to use basic school videotape recording equipment, with one 
camera, a camera tripod, and the built-in camera microphone. No other more elaborate 
video or sound recording equipment will be permitted. In addition, the videotape selected 
for submission may not be edited. The purpose of this policy is to attempt to prevent the 
use of expensive video recording technology by candidates, which might put some 
candidates at a disadvantage in the quality of their videotape submission. 

The above policies may likely be revised as they are reviewed by hundreds of New York State 
educators involved in the assessment development process. 

Future ATS-P Development Activities 

Many ATS-P development activities remain. The ATS-P policies and procedures for 
candidates will undergo careful review and revision. The scoring scale and scoring rubrics 
will be refined and tested. In addition, extensive pilot testing of the ATS-P with New York 
State public school teachers will continue. In all of these activities, the overriding goal will be 
to continue to develop an assessment of teacher performance that will help to ensure that 
permanendy-cerv;ified New York State teachers demonstrate a satisfactory level of teaching 
skills and that the assessment of these skills is reliable and valid. 

The New York State Education Department, through the New York State Teacher Certification 
Examinations, is dedicated to strengthening the standards for admission to the profession of 
teaching. The Assessment of Teaching Skills-Performance is a bold and innovative 
component of a comprehensive assessment system. The ATS-P, with continued refinement as 
it proceeds through a rigorous test development process, may likely serve as a model for 
performance-based teacher certification assessment programs in the ftiture. 
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